®A  125896 


I 


Research  Note  81-18 


CHRONICLE  OF  THE  EVOLUTION  AND 
POTENTIAL  OF  THE  NETMAN  MODEL 


Arthur  I.  Siegel  and  J.  Jay  Wolf 
Applied  Psychological  Services 


TRAINING  RESEARCH  LABORATORY 


U.  $.  Army 


DTIO 


ELECTE 

MAR  2  2  1983 


Research  Institute  for  the  Behavioral  and  Social  Sciences 


May  1981 


Approved  for  qubfic  release;  distnojtien  unlimited. 


PS 


2  f  1*2 


DISCLAIMER 


THIS  DOCUMENT  IS  BEST 
QUALITY  AVAILABLE.  THE  COPY 
FURNISHED  TO  DTIC  CONTAINED 
A  SIGNIFICANT  NUMBER  OF 
PAGES  WHICH  DO  NOT 
REPRODUCE  LEGIBLY. 


UNCLASSIFIED 


SECURITY  CLASSIFICATION  OF  This  PAGE  (Whan  Data  BntaradJ 


REPORT  DOCUMENTATION  PAGE 

READ  INSTRUCTIONS 

BEFORE  COMPLETING  FORM 

EsrTTOamymnTrFf. 

3.  RECIPIENT'S  CATALOG  NUMBER 

*.  TITLE  (and  Subtttta) 

Chronicle  of  the  Evolution  and  Potential  of 
the  NETMAN  Model 

S.  TYPE  OF  REPORT  A  PERIOO  COVERED 

Research  Note 

June  '80-May  '81 

6.  PERFORMING  ORG.  REPORT  NUMBER 

7.  AUTHOR!*) 

8.  CONTRACT  OR  GRANT  NUMBER^*) 

Arthur  I.  Siegel 

J.  Jay  Wolf 

MDA903-78-C-2013 

9.  PERFORMING  ORGANIZATION  NAME  AND  ADDRESS 

Applied  Psychological  Services,  Inc. 

Science  Center 

Wayne,  PA  19087 

10.  PROGRAM  ELEMENT,  PROJECT,  TASK 
AREA  A  WORK  UNIT  NUMBERS 

2Q162717A790 

II.  CONTROLLING  OFFICE  NAME  ANO  ADDRESS 

US  Army  Research  Institute  for  the  Behavioral 

12.  REPORT  OATE 

May  1981 

and  Social  Sciences 

5001  Eisenhower  Ave.,  Alexandria, 

VA  22333 

13.  NUMBER  OF  PAGES 

60  +  viii 

14.  MONITORING  AGENCY  NAME  &  AODRESSCIf  dltlarant  from  Controlling  Otttca) 

19.  SECURITY  CLASS,  (of  thtm  roport) 

UNCLASSIFIED 

15«.  DECLASSIFICATION/ DOWN  GRADING 
SCHEDULE 

IS.  DISTRIBUTION  STATEMENT  (of  tills  fi sport) 

Approved  for  public  release;  distribution  unlimited 

17.  DISTRIBUTION  STATEMENT  (of  th*  mb  m  tract  mntmrmd  In  Block  20,  It  dlftormxt  from  Report) 

18.  supplementary  notes 

If.  KEY  WOROS  (Contlnuo  on  rorormm  old*  It  nmcmoomry  and  Identity  by  block  numbor) 

Computer  Simulation  System  Analysis  Operator  Analysis 

Exercise  Management  Readiness  Human  Factors 

Exercise  Control  NETMAN  Manning 

Human  Performance  Reliability  Exercise  Evaluation  System  Load 

M.  ABSTRACT  (Cmttwna  ma  rararaa  *M»  H  rmeaaaaay  raid  tdanlltr  hr  block  numbar) 

The  NETMAN  computer  simulation  model,  its  development,  evolution,  and 
results  from  its  sensitivity  and  validity  testing  and  application  are 
chronicled.  NETMAN  simulates  message  processing  in  computer-based  military 
field  exercise  control  systems.  In  order  to  present  NETMAN  in  historical 
perspective,  its  genealogy  and  its  parent  model  (MANMOD)  are  described. 

Emphasis,  however,  is  given  to  the  efforts  devoted  to  NETMAN  during  the  past 
three  years  of  research,  which  this  report  also  summarizes. 

00  ,  jSTts  W73  ew-no-  OF  »  hov  «s  is  obsolete  UNCLASSIFIED 


SECURITY  CLASSIFICATION  OF  this  PAGE  (Whan  Data  Bntarad) 


UNCLASSIFIED 


SECURITY  CLASSIFICATION  OF  THIS  PAOE(T»»l«n  DM*  BnltndJ 


The  logic  and  structure  of  NETMAN  are  presented,  and  the  model's  strengths 
and  weakness  are  assessed  in  a  critical  evaluation  of  its  current  status  as  a 
tool. 


UNCLASSIFIED 


SECURITY  CLASSIFICATION  OF  THIS  PAGEfWhxi  DM*  B nffd) 


Chronicle  of  the  Evolution  and  Potential  of  the  NETMAN  Model 


Arthur  I.  Siegel 
J.  Jay  Wolf 


prepared  for 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

Alexandria ,  Virginia  22333 


by 


Applied  Psychological  Services, 
Science  Center 
Wayne,  Pennsylvania  19087 


Inc. 


Accession  For 

NTIS  GRAil 
DT1C  TAR 


Unannounced 


Just if  i  eat  i  c:i _ _ 


under 

Contract  MDA903-78-C-201 3 
May  1981 


By - 

Distribution/ 


Availability  Codes 
.Avail  and/or 


Dlst 


Special 


iii 


EXECUTIVE  SUMMARY 


V 

An  overview  is  presented  of  the  past  efforts,  extending  over  more 
than  five  years,  relative  to  the  NETMAN  computer  simulation  model. 
NETMAN's  goal  is  to  allow  analysis  of  military  field  exercise  control 
systems  through  digital  simulation  techniques.  The  efforts  include  de¬ 
sign  of  the  simulation  model  itself,  extensive  sensitivity  tests,  model 
enhancements,  validity  testing  against  outside  criterion  data,  and  applica¬ 
tion  of  the  model  to  a  new  field  exercise  control  system  concept. 

The  NETMAN  model  is  structured  to  simulate  the  message  han¬ 
dling  aspects  of: 

( l]  field  exercise  data  collection  and  message  coding 
by  referees," 

(  2 1  message  transmission  by  radio  operators  ' 

3;  message  entry  into  and  processing  by  a  control 
computer  including  decoding  and  data  base  up¬ 
date,  <?> 

4 i  receipt  of  the  decoded  message  by  a  controller. 

The  tasks  performed  at  each  level  (plus  delays)  are  simulated  stochas¬ 
tically  and  results  are  generated  in  terms  of  message  processing  effec¬ 
tiveness. 

J 

Objectives 

The  primary  purpose  of  this  report  is  to  summarize  the  past  and 
recent  efforts  using  the  NETMAN  and  its  predecessor  model  (MANMOD) 
so  that  an  objective  evaluation  can  be  made  for  future  action  and/or 
utilization.  This  purpose  is  accomplished  by  overviews  of  the  models  and 
summaries  of  all  efforts  and  results  achieved.  References  to  original 
reports  are  presented  in  the  bibliography.  This  report  is  oriented  pri¬ 
marily  for  those  who  are  not  concerned  with  the  specific  programmatic 
or  functional  details  of  the  model  but  who  want  an  assessment  of  its  val¬ 
ue  and  utility  or  are  interested  in  its  potential  for  future  application. 


Scope  of  Report 

This  report  includes  several  summary  chapters  on  the  NETMAN 
model's  background; 


v 


•  Chapter  1  presents  the  objectives  of  NETMAN,  a 
chronological  summary  of  its  predecessor  model 
(MANMOD),  and  a  similar  historical  summary  of 
NETMAN 

•  Chapter  II  offers  an  overview  of  the  logical  struc¬ 
ture  and  operation  of  both  models. 

These  are  followed  by  overview  chapters  which  review  recent  efforts: 

•  Chapter  III  reports  on  sensitivity  and  validation 
tests,  beginning  in  1977,  conducted  on  NETMAN 

•  Chapter  IV  introduces  a  new  concept  for  a  more 
automated  field  exercise  management  system  than 
is  presently  operational  called  Exercise  Monitoring, 
Assessment,  and  Reporting  System  (EMARS).  This 
chapter  also  reports  on  the  effort  in  which  NETMAN 
was  employed  to  simulate  the  EMARS  system. 

Last,  the  future  of  NETMAN  is  evaluated: 

•  Chapter  V  summarizes  the  experience  to  date  on  the 
applicability  of  NETMAN.  Its  potential  and  its  lim¬ 
itations  are  discussed  and  conclusions  and  recommen¬ 
dations  are  presented. 
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CHAPTER  1 


BACKGROUND 


Objectives  of  NETMAN 

The  NETMAN  computer  simulation  model  was  developed  to  pro¬ 
vide  a  technique  for  economical,  yet  thorough,  investigation  of  alterna¬ 
tives  in  the  design  of  military  field  exercise  control  systems.  NETMAN 
permits  its  users  to  evaluate  the  relative  impact  of  such  factors  as  per¬ 
sonnel  distribution,  system  configuration,  training,  and  workload.  In 
general,  field  exercise  management  systems  perform  one  or  more  of  the 
following  functions; 

•  control  or  management  of  field  exercises 

•  tracking  of  exercise  progress 

•  calculation  of  scores  for  assessment  of  com¬ 
bat  performance  and  proficiency 

•  display  of  combat  and  unit  performance  status 

•  preparation  of  performance  readiness  reports 

•  support  of  after  action  reviews 

Figure  1-1  presents  the  basic  elements  of  a  total  Army  field  ex¬ 
ercise  management  system.  The  figure  is  intentionally  shown  with  such 
generality  as  to  be  useful  in  describing  the  features  of  essentially  any 
monitored  Army  field  exercise.  Figure  1-1  shows  as  unit  "A"  the  friend¬ 
ly  force,  whose  competence  is  to  be  tested,  together  with  their  total  re¬ 
sources.  Their  combat  opponents  (opposing  force)  are  shown  as  unit  "B.  " 
The  designation,  MC,  "  identifies  the  instrumentation  and  measurement 
subsystem.  It  includes  the  totality  of  all  resources  deployed  on  the  exer¬ 
cise  range  or  elsewhere  to  monitor  the  exercise  and  to  report  the  perform¬ 
ance  of  the  forces.  The  components  offering  the  means  by  which  the  com¬ 
munications  are  effected  is  termed  the  communications  subsystem,  "D.  " 
The  communications  subsystem,  provides  the  capability  for  the  instru¬ 
mentation  and  measurement  subsystem  to  report  its  findings  for  analysis. 

The  exercise  control  function  ("E"),  as  its  name  implies,  con¬ 
sists  of  those  personnel  who;  (1)  establish  the  exercise  conditions  and  the 
combat  objectives,  and  (2)  define  the  required  documentation,  procedures, 
directives,  and  resources. 


FRIENDLY  FORCE  OPPOSING  FORCE 


Figure  1-1.  Basic  elements  of  an  exercise  control  and 
management  system. 


2 


The  exercise  monitoring  and  reporting  subsystem  is  designated 
as  "F"  in  Figure  1-1.  This  subsystem  manipulates  data  provided  as  in¬ 
put  to  determine,  display,  and  report  on  status  and  unit  performance,  and 
convey  the  lessons  to  be  learned  to  the  exercise  participants  at  all  levels. 

Field  training  exercises  serve  an  important  role  in  the  training  of 
military  units  for  the  rapid  transition  from  peacetime  readiness  to  war¬ 
time  effectiveness  on  the  modern,  highly  lethal,  dynamic  battlefield.  The 
current  manual  system*  for  conducting  such  exercises  imposes  costly  de¬ 
mands  on  resources  during  the  period  necessary  to  form  and  train  the  ex¬ 
ercise  staff  and  to  plan,  conduct,  and  report  on  the  exercise.  In  addition, 
the  methods  and  techniques  for  maneuver  control,  casualty  and  damage 
assessment,  and  simulation  of  combat  events  are  unwieldy  and  time  con¬ 
suming,  degrading  the  training  effectiveness.  These  aspects  serve  to 
encourage  the  development  of  new  exercise  control  systems.  A  model 
such  as  NETMAN  makes  test  of  such  new  systems  possible  while  the  sys¬ 
tems  are  in  the  conceptual  stage  of  development. 

Based  on  a  general  purpose  model  of  information  processing  sys¬ 
tems,  NETMAN  simulates  selected  operations  (i.  e. ,  staff  duties,  staff 
functions,  and  machine  functions)  as  a  joint  function  of  the  interactions 
between  the  exercise  data  flow,  human  information  processing,  and  other 
sources  of  variance  (e.  g. ,  personnel  characteristics)  to  produce  predic¬ 
tions  of  overall  system  performance.  On-line  inputs  to  the  program  per¬ 
mit  simulation  of  candidate  configurations  that  reflect  changes  in  personnel 
and  machine  functions  (particularly  at  the  man -machine  interface),  and 
level  of  personnel  training  for  comparison  of  alternatives  (i.  e. ,  the  ability 
to  ask  "what  if"  questions  on-line. )  Appendix  A  to  this  repoi't  provides  a 
summary  of  why,  how,  and  when  to  use  the  NETMAN  simulation  model. 


Chronological  Summary  of  NETMAN  and  Precursor  Systems 

This  section  presents  a  historical  perspective  of  the  development, 
testing,  and  utilization  of  two  message  or  communications -oriented  man- 
machine  simulation  models  called  MANMOD  and  NETMAN. 


MANMOD  Model 


Work  on  a  model,  started  in  eaiTy  1972,  led  to  the  MANMOD  pro¬ 
gram  to  simulate  the  details  of  message  processing  within  the  current  field 
exercise  control  system  called  TOS. 


*FM  105-5,  Maneuver  Control,  31  December  1973. 
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TOS  is  an  automated,  secure  information  processing  system  de¬ 
signed  to  assist  military  commanders  and  their  staffs  at  Field  Army, 
Corps,  and  Division  levels  in  the  conduct  ot  tactical  operations.  The 
TOS  is  part  of  an  overall  Integrated  Battlefield  Control  System.  It  serves 
information  collection  and  dissemination  functions --such  as  transmission 
of  orders,  information  requests,  and  practically  any  information  concern¬ 
ing  the  battlefield  situation. 

MANMOD  simulates  the  activities  within  a  single  communication 
station  in  the  TOS  field  army  situation.  The  station  is  manned  by  one  or 
more  G-3  (operations)  officers,  and  one  or  more  action  officers.  These 
men  receive  messages  of  various  types  which  are  delivered  to  the  station 
at  times  unknown  in  advance  of  the  simulation.  Their  function  is  to  screen 
incoming  messages,  decide  on  the  order  in  which  the  messages  are  to  be 
processed,  select  the  appropriate  message  form  to  which  they  transform 
the  content  of  each  message,  and  enter  the  messages  into  the  TOS  data 
bank.  The  model  simulates  the  acts  and  behaviors  of  TOS  personnel  as 
they  receive,  prioritize,  code,  and  enter  messages  into  the  TOS  comput¬ 
er.  The  model  is  completely  general  and  allows  for  the  simulation  of  per¬ 
sonnel  of  different  competency'  and  stress  tolerance,  along  with  a  variation 
in  message  load  and  content.  Output  from  the  model  allows  statement  of 
personnel  error  source,  time  delay,  and  error  type  within  the  processing 
sequence.  The  model  was  designed  for  standard  batch  run  processing  on 
the  CDC  3300  computer. 

In  the  first  stage  of  the  development  of  MANMOD,  the  model's 
output  was  verified  through  a  comparison  with  independently  collected 
criterion  data.  Substantial  correspondence  was  indicated  between  the 
criterion  data  and  the  predictions  of  MANMOD.  Accordingly,  initial  con¬ 
fidence  in  the  validity  of  MANMOD  was  supported.  The  model,  user  in¬ 
structions,  and  sensitivity  testing  were  presented  in  Siegel,  Wolf,  and 
Leahy  (1973). 

In  a  follow-on  effort,  (Siegel,  Wolf,  Leahy,  Bearde,  &  Baker, 
1973),  the  model  was  modified  to  operate  in  an  interactive  time  sharing 
mode,  allowing  the  experimenter  to  interact  in  a  "conversational"  mode 
with  the  model  and  to  enter  data  "on  line.  "  This  interaction,  performed 
through  a  computer  terminal,  greatly  increased  the  ease  with  which  sim¬ 
ulations  could  be  performed.  Various  extensions  of  the  original  model 
were  also  made  at  that  time. 

A  variation  of  MANMOD  was  developed  to  allow  intermingling  of 
real  time  experimental  data  and  computer  generated  simulation  data. 

In  this  variation,  one  or  more  actual  system  operators  process  messages 
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at  one  level  of  the  TOS  system  while  the  computer  simulates  the  rest  of 
the  TOS  (Leahy,  Lautman,  Bearde,  &  Siegel,  1974). 

In  1974,  (Leahy,  Lautman,  Bearde,  &  Siegel,  1974)  the  MANMOD 
was  adapted  for  the  Univac  1108  computer.  The  system-specific  aspects 
of  FORTRAN  were  removed,  and  the  logic  was  modified  and  the  output 
extended.  These  increased  the  realism  of  the  simulation.  In  1975,  the 
MANMOD  was  also  modified  to  exchange  data  with  two  other  Army  com¬ 
puter  models  (CASE  and  SAMTOS)  in  such  a  way  as  to  maximize  the 
strong  points  of  each  of  the  models.  Other  computer  simulation  models 
(CASE  and  SAMTOS)  approach  the  simulation  in  different  ways  and  possess 
different  objectives  from  the  MANMOD.  The  MANMOD  allows  for  the 
study  of  the  effects  of  varying  operator  and  manning  considerations  on 
message  processing  effectiveness.  The  CASE  and  the  SAMTOS  models 
are  more  concerned  with  equipment  functions  with  (more  or  less)  con¬ 
stant  human  operators  in  the  loop. 

During  the  course  of  the  integration  effort,  a  number  of  logic 
changes  (largely  concerned  with  the  simulation  of  error  message  proc¬ 
essing)  were  introduced  into  the  logic  of  the  MANMOD.  The  modifications 
were  tested  for  sensitivity  and  reasonableness  of  effect  on  model  output. 
The  modifications  were  found  to  be  acceptable  from  the  point  of  view  of 
both  of  these  criteria.  It  was  concluded  that  the  body  of  results  on  hand 
supported  contentions  favoring  the  use  of  the  model  for  a  number  of  func¬ 
tions  relative  to  system  design,  training  requirements  and  objectives  de¬ 
rivation,  personnel  requirements,  and  tradeoffs.  A  report  on  the  inte¬ 
gration  (Leahy,  Siegel,  &  Wolf,  1975a)  and  a  user's  manual  (Leahy, 
Siegel,  &  Wolf,  1975b)  were  prepared. 

Additional  detail  on  the  chronology  of  MANMOD' s  development,  en¬ 
hancement  and  testing  are  given  in  Appendix  B. 


Chronological  Summary  -  NETMAN  Model 

In  1975,  development  began  on  the  NETMAN  model  itself.  Its 
main  function  is  to  enable  an  understanding  of  the  underlying  system  as¬ 
pects  of  military  field  exercise  control  systems.  To  maximize  use  of 
existing  modeling  concepts,  some  of  the  elements  included  in  NETMAN 
were  developed  (or  adapted)  from  the  previously  tested  MANMOD.  For 
example,  the  simulation  technique  for  task-by-task  performance  eval¬ 
uation  in  MANMOD  was  used  substantially  intact  in  the  development  of 
NETMAN  (Siegel,  Leahy,  &  Wolf,  1977). 
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Other  features  from  MANMOD  integrated  into  NETMAN  include; 


•  Generation  of  message  queues  at  the  beginning  of 
each  simulation  iteration 

•  Simulation  of  the  effects  of  operator  factors,  such 
as  speed  and  precision,  on  performance 

•  Stochastic  variation  (i.  e. ,  random  combinations  of 
chance  events)  to  increase  the  generality  of  the  out¬ 
put.  This  variation  requires  that  a  simulation  be  re¬ 
peated  a  number  of  times  and  the  results  averaged 

in  order  to  produce  a  stable  output 

•  Automatic  listing  of  inputs  and  run  summaries  with 
various  additional  outputs  available  as  options. 

NETMAN  simulates  individually  each  person  and  each  message 
involved  within  a  semiautomatic  technique  for  evaluating  field  exercises. 
Information  collection,  message  preparation,  message  handling,  mes¬ 
sage  processing,  and  message  evaluation  are  included  within  the  sim¬ 
ulation.  The  personnel  involved  in  a  simulation  may  include  up  to  27 
referees,  27  radio  operators,  and  three  controllers  interacting  in  a  fix¬ 
ed  network  of  communications  lines  while  sharing  time  on  a  central  com¬ 
puter. 


In  addition  to  an  overall  measure  of  performance,  the  NETMAN 
provides  four  component  measures  on-line  and  detailed  printouts  are  made 
available  off-line.  The  detailed  message  processing  output  shows  the  de¬ 
tails  of  the  results  of  the  simulation  of  each  task  in  the  processing  of  mes¬ 
sages. 


An  hourly  summary  is  provided  which  presents  a  consolidation  of 
the  results  of  each  simulated  hour's  work  across  all  iterations  and  includes 
items  such  as;  number  of  messages  completed,  time  spent  working,  end 
of  hour  stress  level,  performance  and  aspiration,  time  spent  performing 
various  processes,  and  average  time  per  message. 

A  simulation  run  summary,  produced  after  N  iterations  of  the 
exercise,  includes  manpower  utilization,  message  processing  times,  over¬ 
all  effectiveness  indicators,  and  workload  summary  information. 
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A  full  description  of  NETMAN  is  contained  in  Siegel,  Leahy,  & 
Wolf  (1977)  together  with  a  discussion  of  model  utilization,  program  flow 
charts,  subroutine  definitions,  user  input-output  formats,  and  task  anal¬ 
yses. 


Initial  NETMAN  Sensitivity  Tests 

The  1977  report  also  described  the  methods  and  results  of  a  set 
of  sensitivity  tests  of  the  model.  These  assessed  the  rationality  and 
sensitivity  of  the  NETMAN  model  to  input  variation.  While  these  tests 
provided  no  information  relative  to  the  predictive  validity  of  the  model, 
the  results  allowed  initial  evaluation  of  the  model  from  the  point  of  view 
of  the  reasonableness  of  its  output.  Within  the  tests,  four  input  variables 
were  manipulated  (operator  speed,  operator  precision,  message  fre¬ 
quency,  and  message  length).  The  results  indicated  that; 

On  the  general  level,  the  sensitivity  tests  indicated 
reasonable  model  output  directional  response  to  input  var¬ 
iation.  However,  while  sensitivity  tests  results  generally 
supported  the  rationality  of  the  results,  some  problem 
areas  were  evidenced.  The  principal  problem  involved  an 
indication  of  excessive  queuing  time  because  of  a  logic  er¬ 
ror  relative  to  the  time  synchrony  of  queuing  aspects  rel¬ 
ative  to  the  simulated  computer.  This  error,  naturally 
also  exerted  effects  on  the  various  individual  effectiveness 
indices.  Subsequent  to  the  initial  sensitivity  test  runs, 
this  logic  was  corrected  and  a  more  satisfactory  set  of  re¬ 
sults  were  obtained. 

. .  .the  results  of  the  sensitivity  tests  suggest  that  the 
data  obtained  from  NETMAN  are  directionally  logical  and 
that  the  output  is  sensitive  to  input  variation.  To  this  ex¬ 
tent,  a  workable  model  can  be  said  to  have  been  achieved 
(Siegel,  Leahy,  &  Wolf,  1977). 

In  1977,  a  three  year  program  into  NETMAN  enhancements,  test¬ 
ing,  utilization,  and  evaluation  was  initiated.  Initially,  the  model  was  the 
subject  of  both  sensitivity  and  validation  testing.  A  series  of  60  computer 
runs  enabled  statistical  test  of  the  effects  of  a  variety  of  personnel  and 
workload  variables,  manpower  configurations,  and  task  variables.  The 
results  were  found  to  be  reasonable  and  appropriate;  the  most  influential 
variables  were  operator  speed,  operator  precision,  and  network  config¬ 
uration.  The  psychological  factors  (stress,  aspiration  level)  exerted  a 
much  less  powerful  effect  on  output.  The  testing  and  validity  analysis  is 
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summarized  in  Chapter  3  of  this  report  and  is  described  in  full  in 
Siegel,  Leahy,  &  Wolf  (1979). 


■» 


Reports 


Appendix  C  contains  a  table  which  lists  the  technical  reports 
which  have  resulted  from  the  efforts  associated  with  the  two  models. 
In  addition  to  the  title,  author(s),  and  date  of  each  report.  Appendix  C 
also  shows  the  task  and  number  of  men  simulated  in  each  application. 
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CHAPTER  2 


OVERVIEW  OF  STRUCTUR  E  OF  MANMOD  AND  OF  NET  MAN 


Chapter  2  contains  an  overview  of  the  structure  and  operation  of 
both  the  MANMOD  and  the  NETMAN  models.  The  presentation  begins 
with  a  comparison  of  differences  and  a  summary  of  common  features, 
followed  by  a  brief  review  of  the  logic  of  the  models  themselves. 

Capability  Comparison 

Both  the  MANMOD  and  NETMAN  models, implemented  in  the  form 
of  FORTRAN  IV  computer  programs,  keep  track  of  each  message  proc¬ 
essed  in  the  simulated  system.  They  simulate  the  acts  and  behaviors  of 
operations  personnel  as  they  receive,  prioritize,  code,  and  enter  these 
messages  into  the  system.  The  models  are  general  in  that  they  allow  for 
the  simulation  of  personnel  of  different  competencies  and  stress  toler¬ 
ances,  along  with  a  variation  in  message  load  and  content. 

These  models  combine  the  effects  of  such  features  as  message 
generation  and  queuing,  detailed  message  processing  procedure,  error 
rates,  and  personnel  characteristics,  along  with  stochastic  variations  to 
yield  predictions  of  system  performance.  The  basic  nature  of  both  mod¬ 
els  is  stochastic  and,  as  a  result,  a  number  of  repetitions  is  required  to 
produce  a  stable  result.  Repetitions  of  a  simulation  with  the  same  set  of 
input  variable  conditions  are  called  a  run.  At  the  conclusion  of  a  run,  the 
run  summary  (integrating  results  of  various  repetitions)  is  produced. 

Along  with  the  simulation  of  message  processing  by  operators,  the 
models  include  the  simulation  of  the  computer  embedded  in  the  processing 
system.  Global  information  about  both  of  the  message  processing  models 
is  given  in  Table  2-1.  Both  models  handle  multiple  message  types  and 
varying  message  priorities. 


MANMOD  Model  Overview 


The  MANMOD  model  combines  the  effects  of  such  features  as 
message  queuing,  detailed  message  processing  procedure,  error  rates, 
and  personnel  characteristics,  along  with  stochastic  variations  to  yield 
predictions  of  system  performance.  Parameters  which  may  be  varied  to 
study  their  impact  on  system  performance  include: 
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Table  2-1 


Description  of  Message  Handling  Models 


System  Simulated 

Tactical  Operational 

System 

Military  Exercise  Control/ 
Evaluation  System 

Program  Name 

MANMOD 

NETMAN 

Maximum  Number  of  Men 
Simulated 

6 

57 

Types  of  Personnel 

2 

4 

Major  Input  Parameters 

Shift  Length 

Number  of  Personnel 

Error  Rates 

Operator  Characteristics 
Speed 

Precision 

Aspiration 

Stress 

Message  Characteristics 

Shift  Length 

Number  of  Personnel 

Error  Rates 

Operator  Characteristics 
Speed 

Precision 

Aspiration 

Stress 

Message  Characteristics 
Networks 

Major  Output 

System  Effectiveness 

Time  Worked 

Operator  Stress,  Aspira¬ 
tion 

Message  Processing  Sta¬ 
tistics 

Errors 

System  Effectiveness 

Time  Worked 

Operator  Stress,  Aspira¬ 
tion 

Message  Processing  Sta¬ 
tistics 

Development  Initiated 

1972 

1975 

Programming  Language 

FORTRAN  IV 

FORTRAN  IV 
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•  hours  per  shift 

•  number  of  action  officers 

•  number  of  input-output  device  operators 

•  operator  fatigue 

•  error  rates  (per  hour,  per  type  of  effort, 
per  type  of  message) 

•  personnel  characteristics  (speed,  precision, 
aspiration,  and  stress  threshold) 

•  message  arrival  frequency 

•  message  workload 

•  message  type  mix 

•  message  length 

•  message  handLing  procedure 

There  are  up  to  four  sequences  of  task  elements  provided  to  re¬ 
present  the  tasks  executed  by  an  operator  in  performing  his  duties.  Each 
sequence  has  a  capacity  of  up  to  20  task  elements.  The  model  handles 
up  to  6  men  of  2  types  (any  combination  of  Action  Officers  and  Input-Output 
Device  [UIOD]  operators),  4  types  of  operator  errors,  7  types  of  mes¬ 
sage,  5  message  priority  classifications,  and  a  shift  length  of  up  to  12 
hours. 


One  class  of  simulated  messages  is  "generated"  by  the  MANMOD 
at  random  times.  The  other  class  involves  situation  reports  which  are 
assumed  to  arrive  only  during  the  last  quarter  of  a  given  hour.  To  proc¬ 
ess  both  classes  of  message,  the  performance  of  one  or  more  UIOD  op¬ 
erators  is  simulated.  Working  from  the  queue  of  message  forms  pre¬ 
pared  by  the  officers,  the  UIOD  operators  enter  each  message  into  the 
Tactical  Operations  System  (TOS)  using  a  keyboard  and  CRT  edit/verify 
device.  The  purpose  of  the  model  is  to  simulate  these  personnel  on  an 
hour-by-hour  basis  over  a  single  work  shift.  MANMOD  treats  the  above 
general  task  assignment  to  operators  and  the  sequencing  thereof  as  soft, 
i.  e. ,  since  they  are  controlled  by  input  data,  they  may  be  readily  changed. 

Structure  of  the  MANMOD 

Figure  2-1,  taken  from  Siegel,  Wolf,  and  Leahy  (1973)  shows 
the  subroutines  and  their  interrelation  with  MANMOD's  main  processing 
flow.  Following  data  read -in,  there  is  an  optional  recording  of  the  in¬ 
put  data.  Conditions  are  then  reset  (circle  b)  for  the  start  of  simulation 
for  a  new  TOS  shift.  The  backlog  subroutine  (BAKLOG)  generates  data 
representing  messages  in  the  Action  Officer's  "in-box"  at  the  start  of 
each  iteration.  At  circle  c,  following  reset  of  counters  and  registers  for 
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Figure  2-1.  Summary  of  initial  MANMOD  flow  logics  (from  Siegel, 
Wolf,  &  Leahy,  1973). 
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record  keeping  of  hourly  results,  the  message  generation  subroutine  de¬ 
velops  data  representing  messages  which  will  arrive  during  the  coming 
hour.  These  are  merged  with  the  backlog  in  order  by  time  of  arrival, 
and  the  contents  of  this  hourly  message  queue  are  optionally  recorded. 

At  circle  d,  the  processing  of  each  message  in  turn  by  a  single  operator 
(either  G-3,  Action  Officer,  or  UIOD  operator)  begins.  The  Man  Deter¬ 
mination  Subroutine  selects  the  appropriate  man  to  simulate  next,  and 
determines  the  next  available  appropriate  message  for  this  selected  man 
to  process.  The  operator  stress  and  aspiration  conditions  applicable  to 
that  operator  message  situation  are  calculated  next.  At  this  point  (cir¬ 
cle  j),  all  data  are  available  for  the  detailed  task  element -by-task  ele¬ 
ment  simulation  for  the  message  and  operator  selected.  This  is  accom¬ 
plished  by  the  Operator  Processing  Subroutine  which  manipulates  mis¬ 
sion  task  analysis  data.  During  this  subroutine,  the  detailed  results  of 
the  simulation  of  the  performance  of  each  task  element,  as  well  as  the 
summary  results  for  this  one  message,  may  be  optionally  recorded  for 
printing. 

Following  this,  queues  are  adjusted  and  the  cycle  goes  back  to 
circle  d  for  processing  of  all  messages  which  can  be  handled  before  the 
next  one  hour  segment  is  completed.  When  the  end  of  an  hour  condition 
is  reached  for  all  men,  the  results  of  the  hour's  simulated  activity  are 
optionally  printed.  The  process  of  simulating  in  one  hour  segments  is 
repeated  (back  to  circle  c)  until  the  simulation  of  one  iteration  of  an  en¬ 
tire  shift  is  completed.  When  a  shift  iteration  is  finished,  the  iteration 
summary  subroutine  generates  summary  data  over  the  shift.  The  pro¬ 
cess  is  repeated  (back  to  circle  b)  for  each  iteration  and  after  all  iter¬ 
ations  are  completed,  the  run  summary  subroutine  summarizes  and  re¬ 
cords  all  the  pertinent  performance  figures  to  complete  a  simulation  run. 
Multiple  runs  are  processed  sequentially  through  the  entire  process  by 
returning  to  the  input  subroutines  at  circle  a  and  processing  continues  for 
as  many  sets  of  data  as  are  provided  as  input. 

Each  computer  run  of  the  model  represents  a  simulation  of  up  to 
10  hours  in  duration,  in  which  up  to  2000  messages  can  be  processed.  In 
this  model,  each  operator  type  has  its  own  task  analysis. 

The  end  result  is  the  ability  to  answer  questions  such  as; 

•  How  does  system  effectiveness  vary  as  a  function 
of  message  load,  operator  level  of  aspiration,  mes¬ 
sage  arrival  time  distribution,  or  personnel  pro¬ 
ficiency? 
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•  What  is  the  effect  of  increasing  or  decreasing  the 
manning  level  or  personnel  proficiency? 

•  How  much  stress  is  on  the  operators  during  the 
performance  of  the  work  of  each  hour? 

•  What  is  the  error  rate  for  various  message  types 
and  for  various  mannings  and  personnel  attributes 
within  each  manning? 

•  How  much  time  is  spent,  on  the  average,  proc¬ 
essing  each  type  of  message? 

Overview  of  the  NETMAN  Model 


The  NETMAN  model  was  designed  to  allow  simulation  of  message 
processing  in  a  system  composed  of  up  to  three  networks.  Each  network 
may  be  composed  of  up  to  nine  referees,  up  to  nine  radio  operators,  and 
one  controller.  One  computer  is  assumed  to  accommodate  the  three  net¬ 
works.  As  such,  the  model  allows  simulation  and  test  of  the  effects  on 
system  effectiveness  of  varying  such  aspects  as:  number  of  referees, 
number  of  networks,  task  procedures,  message  arrival  rate,  message 
length,  and  operator  characteristics. 

The  results  of  the  simulation  are  interpretable  in  terms  of  a  num¬ 
ber  of  formal  effectiveness  measures  (accuracy,  thoroughness,  respon¬ 
siveness,  completeness)  and  an  overall  effectiveness  index.  Additionally, 
the  results  are  interpretable  in  terms  of  such  model  results  as  work  time, 
stress  imposed,  message  processing  time,  errors,  number  of  messages 
processed,  and  fatigue. 

Structure  of  the  NETMAN  Model 


The  overall  flow  of  the  field  exercise  data  processing  situation  is 
configured  as  a  message  processing  throughput  as  shown  in  Figure  2-2. 
The  path  of  information  transmittal  is  primarily  from  left  to  right  in  the 
diagram.  The  far  left  column,  containing  flags  numbered  from  1  to  27, 
represents  available  field  situation  information.  Based  on  this  informa¬ 
tion,  his  awareness  of  it,  and  his  interpretation  of  it,  the  referees  (Rj  to 
II27)  encode  messages  for  transmittal.  The  procedure  followed  by  the 
referees  in  the  performance  of  their  duties  is  detailed  in  a  task  analysis 
(TAR).  The  resulting  messages  are  delivered  to  radio  operators  (ROj  to 
RO27)  who  process  the  messages  and  pass  them  to  the  CCC.  The  CCC 
decodes  the  message,  instantly  uses  the  information  to  update  various 
status  tables,  and  makes  the  decoded  messages  sequentially  available  to 
the  field  exercise  controller  (CON).  The  procedure  followed  by  the  com¬ 
puter  is  represented  by  (TAC)  while  the  controller's  tasks  are  shown  by 
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TAN.  As  part  of  his  task,  the  controller  may  communicate  verbally  with 
the  referees.  This  communication,  as  simulated  by  NETMAN,  is  unidi¬ 
rectional  and  does  not  include  interactive  aspects. 

The  respresentation  of  this  situation,  as  simulated  in  the  NET- 
MAN  model,  is  shown  in  Figure  2-3.  A  full  description  of  the  detailed 
internal  logic  of  the  model  and  the  data  flow  within  the  model  is  present¬ 
ed  in  Leahy,  Siegel,  and  Wolf  (1980). 

As  shown  in  Figure  2-3,  processing  proceeds  within  the  model  in 
segments.  Each  segment  is  completed  before  the  succeeding  segment  is 
activated.  For  example,  in  block  2.  0  all  of  the  messages  which  will  be 
available  to  be  processed  during  a  simulation  are  generated  before  sim¬ 
ulation  begins.  After  generation  of  the  messages,  control  passes  to  sub¬ 
routine  REFRE.  After  completion  of  the  processing  of  one  message  for 
each  referee,  subroutine  RADOP  becomes  operational.  In  RADOP  each 
radio  operator  processes  one  message.  After  each  radio  operator  is 
finished,  the  messages  are  sorted  for  proper  arrival  time  to  the  comput¬ 
er  and  then  sent  to  subroutine  CCC  which  processes  each  message.  After 
the  computer  has  completed  each  message,  control  passes  to  subroutine 
CONTR.  CONTR  simulates  the  processing  of  the  messages  by  the  con¬ 
trollers.  Note  that  the  processing  of  each  message  is  not  performed  in 
true  time  synchrony  with  the  simulated  system.  Instead,  each  message 
is  tagged  with  simulation  processing  times  and  delay  times,  which  oc¬ 
curred  during  the  processing.  Since  each  message  is  independently  proc¬ 
essed,  time  synchrony  is  not  needed  and  does  not  affect  the  fidelity  of 
the  output. 

NETMAN's  limits  are  shown  in  Table  2-2.  Some  of  these  limits 
can  be  changed  by  the  user.  However,  since  such  changes  require  a  large 
number  of  changes  throughout  the  program,  these  changes  should  not  be 
considered  trivial.  Other  limits,  such  as  the  range  of  values  for  operator 
speed  or  operator  precision,  are  based  on  experience  in  the  use  of  NET- 
MAN.  The  use  of  values  outside  these  limits  violates  the  validated  and, 
thereby,  trustworthy  use  of  the  model.  Violation  of  some  of  these  limits 
(e.g.,  use  of  operator  precision  values  in  excess  of  1.1)  may  result  in  a 
perpetual  loop  within  a  simulation. 
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INPUT 

CARO  INPUT 
EXPEB INPUT 
INITIALIZATION 


MESGEN 

MESSAGE  GENERATION 
FOR  ALL  MESSAGES 
OF  ITERATION 


REFRE 

PROCESS  A  MESSAGE 
FOR  EACH  REFEREE 
(CALL  PRQC1 


RADOP 
PROCESS  A 
MESSAGE  FOR 
EACH  RADIO 
OPERATOR 
(CALL  PROC) 


SET  MESSAGES  INTO 
BUFFER  WAITING 
AREA 
IBUFF} 


CCC 

PROCESS  MESSAGES 
DEPENDING  ON 
MESSAGE  LOAD 
(CALL  P  ROC) 


CONTR  CONTROLLER 
PROCESS  MESSAGES 
FROM  COMPUTER 


IS 

ITPHATION 


COMPLETE? 


COMPUTE  OVERALL 
SUMMARY 
STATISTICS 
RUNSUM 


Table  2-2 


Allowable  Range  for  Variables  in  NETMAN  Model 


Function 

Lower 

Limit 

Upper 

Limit 

Number  of  hours 

1.0 

10.0 

Number  of  network  levels 

4.0 

4.0 

Number  of  task  elements  in  each  task  analysis 

1.0 

20.0 

Number  of  message  types 

1.0 

7.0 

Number  of  task  analyses 

1.0 

20.0 

Number  of  networks 

1.0 

3.0 

Number  of  time  segments  on  which  counts  can 
be  maintained 

- 

19.0 

Number  of  referee/ radio  operator  teams  per 
controller 

1.0 

9.0 

Operator  speed 

0.5 

1.5 

Operator  precision 

0.9 

1.  1 

Operator  stress  threshold 

2.0 

9.0 

Operator  level  of  aspiration 

0.5 

1.0 

Days  worked 

1.0 

10.0 
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CHAPTER  3 

SENSITIVITY  AND  VALIDATION  TESTING 


Independent  Review  Panel 

After  its  initial  development,  the  NETMAN  model  was  the  subject 
of  sensitivity  test,  enhancement,  and  validation. 

Before  proceeding  with  the  NETMAN  sensitivity  tests  and  valida¬ 
tion,  an  independent  review  of  the  model  was  sought.  A  review  panel  of 
independent  personnel,  expert  in  various  areas  of  digital  modeling  and 
related  fields,  was  brought  together.  The  panel  was  composed  of  persons 
from  university,  governmental,  and  consulting  organizations.  Disciplines 
represented  on  the  panel  were  computer  programming,  mathematics,  sys¬ 
tems  analysis,  simulation,  modeling,  and  operations  research.  The  panel 
made  suggestions  relative  to:  model  improvement,  additional  features  to 
be  added,  features  which  should  be  deleted,  features  which  may  yield  mis- 
impressions,  errors  in  the  work,  and  attributes  which  are  useful.  Par¬ 
ticularly  important  were  suggestions  relative  to  improvements  concerning 
documentation,  sensitivity  test  considerations,  and  validation. 

The  panel  found  the  model  to  be  highly  portable,  being  written  in 
FORTRAN  IV  with  almost  no  machine  specific  aspects.  (The  exception 
is  the  random  number  generator;  such  routines  are  usually  uniquely  de¬ 
veloped  for  each  computer  system.  )  The  computer  program  was  held  to 
follow  the  rules  of  structured  programming.  Each  major  function  in  the 
program  is  separated  into  its  own  subroutine.  This  allows  program 
changes  to  be  made  relatively  quickly.  The  subroutines  were  said  to  be 
well  organized  and  the  interfaces  between  subroutines  to  be  clearly  iden¬ 
tified.  Internal  documentation,  including  comment  cards,  was  held  to  be 
adequate.  The  fixed  format  input  was  said  to  be  difficult  for  the  user  and 
substitution  of  free  format  input  or  more  identifying  information  was  sug¬ 
gested.  The  panel  also  noted  that  the  program  was  relatively  large  and 
that  the  stress  function  possessed  a  discontinuity. 

The  panel's  suggestions  for  logic  changes  included  the  need  for 
entering  human  parameters  for  each  man  individually  and  for  the  inclusion 
of  stress,  aspiration,  and  fatigue  in  the  error  generation  formula.  The 
panel  also  suggested  different  fatigue  curves  for  different  personnel  types 
since  it  thought  that  the  referees  and  the  radio  operators  in  the  field  may 
fatigue  more  quickly  than  controllers. 
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The  panel  also  recommended  that  the  capability  be  added  for  sim¬ 
ulating  referee  mobility,  simultaneous  referee  observation  of  multiple 
situations,  and  referee  decision  making  functions.  Originally  the  referee 
was  simulated  only  in  his  message  originating  aspect. 

Fluctuation  of  level  of  aspiration  due  to  local  effects  such  as  pres- 
\  ence  of  officers  and  situational  variables  was  indicated  by  the  panel  to  be 

an  additional  desirable  feature  along  with  an  increase  in  the  simulation 
duration  capability  beyond  10  hours. 

A  list  was  compiled  of  the  22  major  issues  raised  during  the  pan¬ 
el  discussion  and  each  of  the  six  panel  participants  was  asked  to  judge 
each  item  on  a  five  point  scale.  After  priorities  were  assigned  by  each 
panel  expert  to  each  item,  the  mean  priority  was  calculated  for  each  item 
and  the  mean  priority  ratings  were  ranked.  The  items,  the  ranking  of  the 
priorities,  and  an  analysis  of  these  22  issues  were  included  in  Applied 
Psychological  Services'  report  (Siegel,  Leahy,  &  Wolf,  1979)  which  de¬ 
scribed  this  effort. 

Three  items  were  tied  for  first  place  in  priority  among  the  panel 
suggestions.  All  of  these  were  accomplished  in  subsequent  work: 

User  manual  enhancement 

Programming  changes  for  improved  ease  of  use 

Discrepancies  between  manual  and  program 

The  fourth  ranked  item  was  utility  testing,  i.  e.  ,  the  use  of  the 
model  in  such  a  way  as  to  draw  out  any  problems,  limitations,  or  other 
unknown  program  characteristics.  This  testing  was  completed  in  sub¬ 
sequent  work  (see  Chapter  4  of  this  report). 

The  next  four  items  were  tied  in  order  of  priority:  norm  devel¬ 
opment,  availability  of  input  data,  data  base  development,  and  abbreviated 
user  mode  (input/ output  data  reduction  program). 

Next,  also  tied  in  priority  order,  were  human  effects  on  error 
rate,  inquiry  mode  for  output,  and  overall  processing.  The  human  effects 
which  might  be  expected  to  affect  error  generation  are  stress,  aspiration, 
and  fatigue.  The  inquiry  mode  would  allow  an  on-line  experimentor  to  re¬ 
quest  the  specific  sets  of  data  or  analyses  which  are  relevant  for  his  use. 
Overall  processing  refers  to  the  entire  throughput  and  the  event  oriented 
sequencing  of  message  handling. 
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Next  ranked  was  the  expanded  simulation  of  referee  activity  de¬ 
scribed  earlier  under  logic  changes.  Some  lower  priority  issues  in¬ 
cluded;  graphic  output  to  allow  the  capability  of  automatically  transform¬ 
ing  the  model  output  into  bar  graphs  or  line  graphs,  fatigue  curve  individ¬ 
ualization  to  allow  different  decrement  curves  for  different  levels  of  the 
network  as  well  as  different  decrements  for  different  operations  at  the 
same  level  of  the  network,  error  frequency  modification  to  allow  stress 
to  affect  the  number  of  errors,  catastrophic  failure  indication  to  allow 
consider  ation  of  those  factors,  such  as  radio  failure,  which  would  actually 
shut  down  or  greatly  reduce  the  effectiveness  of  the  system,  and  a  statis¬ 
tical  package  interface  so  as  to  facilitate  data  reduction. 


Model  Preparation  for  Sensitivity  Tests 

To  facilitate  the  sensitivity  and  validation  test,  a  series  of  fine 
tuning  and  readiness -type  adjustments  was  made  to  the  program.  These 
inc  luded; 

•  new  control  language  files  to  facilitate  sim¬ 
ulations 

•  input  format  expansion  to  allow  entry  of  the 
day  number  for  fatigue  calculation  for  each 
simulated  person  rather  than  a  single  value 
for  all  individuals 

•  input  format  change  to  allow  input  of  different 
values  of  speed,  precision,  aspiration,  and 
stress  threshold  for  each  person  simulated 

•  message  origin  identification  revision  so  that  all 
messages  originating  from  the  same  stimulus 
message  are  keyed  to  that  source  message 

•  addition  of  a  new  type  of  task  element,  random 
walk  task,  to  better  simulate  decision  tasks. 

Associated  with  this  a  new  subroutine,  INKAN, 
was  added  to  control  random  walk  data  input 

•  random  number  generation  correction  to  pre¬ 
vent  its  misuse 

•  variable  dimension  statement  modifications  to 
accommodate  larger  data  arrays  for  fatigue, 
speed,  precision,  aspiration,  and  stress  thresh¬ 
old 
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modification  to  allow  simulation  of  networks 
which  include  one  referee,  one  radio  operator, 
and  one  controller --a  case  not  previously  han¬ 
dled  by  NET  MAN 


®  new  files  for  input,  output,  and  mapping,  compil 
ing,  and  printing  elements.  These  changes  facil 
itated  performance  of  multiple  simulation  runs 


•  enhancements  to  enable  the  analyst  to  enter  task 
difficulty  and  task  duration  data  from  the  terminal 
in  the  interactive  mode. 


Sensitivity  Testing  Methods  and  Results 

A  series  of  more  than  60  computer  runs  was  completed  during 
197  9  to  test  the  performance  of  the  NETMAN  computer  model  under  many 
different  simulated  conditions.  The  effect  of  these  condition  changes  was 
then  carefully  evaluated  from  the  viewpoints  of  rationality  of  output,  cost, 
ease  of  use,  and  program  logic  error  (Siegel,  JLeahy,  &  Wolf,  1979). 

The  first  NETMAN  aspect  investigated  was  output  stability  as  a 
function  of  number  of  iterations.  In  a  stochastic  computer  model  (like 
NETMAN),  which  can  simulate  numerous  combinations  of  likely  and  un¬ 
likely  events,  a  number  of  simulations  of  the  same  mission  is  required  in 
order  to  arrive  at  a  stable  estimate  of  the  output  parameter.  If  an  insuffi 
cient  number  of  repetitions  is  used,  the  output  could  be  sensitive  to  un¬ 
likely  occurrences  and  may  be  biased.  With  more  repetitions,  however, 
the  effec  t  of  unlikely  events  tends  to  balance  out.  The  more  complex  the 
mission  simulated,  the  larger  the  number  of  repetitions  which  are  re¬ 
quired  to  produce  stable  output.  A  relatively  basic  mission  scenario  was 
used  in  these  tests.  The  results  indicated  considerable  results  stability 
with  a  limited  number  of  iterations. 

Parameters  varied  in  the  major  sensitivity  tests  were:  operator 
speed,  operator  precision,  operator  level  of  aspiration,  operator  stress 
threshold,  operator  fatigue  level,  number  of  operator  networks,  number 
of  referee/ radio  operator  teams  per  network,  undetected  error  probabil¬ 
ity,  message  frequency,  message  length,  transmission  delay,  and  task 
difficulty.  The  results  which  were  analyzed  from  such  points  of  view  as 
reasonableness,  meaniugfulnoss,  utility,  dependability,  and  reliability 
generally  indicated  support  for  the  structural  logic  and  internal  validity 
of  the  model. 

The  following  quotes  from  Siegel,  Leahy,  Wolf  (1979)  arc  repre 
sentative  of  the  results  reported  from  the  sensitivity  tests: 
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(1)  As  expected,  in  all  cases,  improved  speed  and 
precision  resulted  in  reduced  message  process¬ 
ing  time. 

(2)  A  strong  trend  is  shown  for  the  percentage  time 
busy  to  increase  as  the  simulated  personnel  be¬ 
came  slower  and  less  precise. 

(3)  Statistical  significance  at  the  0.001  level  was  ob¬ 
served  for  all  three  operator  types  in  both  the  ef¬ 
fect  of  fatigue  on  performance  for  the  mission  time 
periods  and  for  the  hourly  (day)  periods.  This  in¬ 
dicates  the  clear  and  significant  impact  of  fatigue 
on  operator  performance. 

(4)  A  one-way  analysis  of  variance  indicated  the  ef¬ 
fect  [of  undetected  error  probability]  on  accuracy 
to  be  statistically  significant  at  the  0.001  level. 

(5)  The  expected  effect  was  observed  in  message 
processing  (messages  completed)  as  a  function  of 
number  of  characters. 

(6)  ...confirms  the  anticipated  effects  of  changes  in 
task  difficulty  on  the  number  of  messages  per  hour. 

Here,  the  effect  of  increased  messages  processed 
as  tasks  become  less  difficult,  is  clearly  noted  for 
all  operators. 

Discussion 

The  major  result  of  the  sensitivity  tests  was  confirmation  that  the 
NETMAN  model  produces  output  which  varies  in  a  reasonable  manner, 
with  input  variation.  The  most  important  variables  were  found  to  be 
operator  speed,  operator  precision,  and  network  configuration.  The  re¬ 
sults  of  the  sensitivity  tests  suggested  that  the  data  obtained  from  NET- 
MAN  are  directionally  logical  and  that  the  output  is  sensitive  to  input  var¬ 
iation.  To  this  extent,  a  workable  model  was  said  to  have  been  achieved. 

From  the  point  of  view  of  operator  performance  analysis,  the  mod¬ 
el  was  reported  to  be  highly  flexible.  The  analyst  may  vary  operator  char¬ 
acteristics  and,  through  task  analytic  input,  the  characteristics  of  the 
work  per  formed  by  each  operator'.  Accordingly,  a  wide  variety  of  "exper¬ 
iments"  may  be  performed  bj  the  analyst  to  determine  optimum  conditions. 

Another-  r  esult  of  the  sensitivity  tests  was  an  evaluation  of  the  ease 
of  use  and  the  cost  of  use  of  the  NETMAN  computer  model.  In  terms  of 
cost  of  use,  the  NETMAN  program  was  found  to  be  very  efficient  since, 


in  most  cases,  an  extensive  number  of  iterations  is  not  required.  The 
findings  relative  to  ease  of  use  were  equally  satisfactory.  Setup  for  a 
new  simulation  was  estimated  as  not  more  than  40  man  hours,  for  an 
experienced  analyst.  This  time  involvement  was  said  to  depend,  to  some 
extent,  on  the  complexity  of  the  system  being  simulated.  Once  a  simula¬ 
tion  has  been  organized,  however,  changes  and  rerunning  the  simulation 
were  said  (Siegel,  Leahy,  &  Wolf,  1979)  to  be  readily  accomplished. 

NETMAN  Model  Validation 

Following  sensitivity  testing,  a  formal  validation  of  the  NETMAN 
model  was  completed.  The  purpose  of  the  validation  was  to  compare  the 
model's  predictions  with  the  performance  of  an  actual  system. 

To  determine  the  predictive  validity  of  the  NETMAN  model,  the 
Marine  Corps'  Tactical  Warfare  Simulation,  Evaluation,  and  Analysis 
System  (TWSEAS)  was  observed  during  control  of  a  full  battalion  field 
exercise.  Field  observers,  assigned  to  troop  units,  made  measurements 
of  message  processing  time  and  frequency.  Additional  observers  collect¬ 
ed  data  in  the  control  center  on  message  handling.  In  this  manner,  data 
were  obtained  concerning  the  quality  of  the  operation  of  the  TWSEAS  sys¬ 
tem.  Prior  to  the  exercise,  the  control  system  personnel  involved  in  the 
TWSEAS  operation  during  the  field  exercise  were  tested  to  provide  per¬ 
sonnel  operating  characteristics  for  input  to  the  model. 

The  NETMAN  computer  model  was  then  run  to  simulate  the  TWSEAS 
performance  during  the  field  exercise  and  the  NETMAN-generated  data 
were  compared  with  the  criterion  data  (actual  TWSEAS  operation). 

The  actual  performance  data,  called  criterion  data,  were  carefully 
collected  to  represent  as  close  an  approximation  as  possible  to  the  situ¬ 
ation  that  the  model  was  built  to  simulate. 

As  the  result  of  a  criterion  analysis,  eight  TWSEAS  criterion  var¬ 
iables  were  selected  for  assessing  the  validity  of  NETMAN:  thoroughness, 
responsiveness,  total  message  processing  time,  referee  message  proc¬ 
essing  time,  radio  operator  message  processing  time,  controller  message 
processing  time,  transmission  delay,  and  overall  effectiveness. 

Field  Exercise 

In  the  field  exercise,  a  full  Marine  Corps  battalion  landing  team 
assaulted  a  beach  with  armor  and  mechanized  units.  The  objec  tive  of  the 
exercise?  was  to  rescue  an  abducted  member  of  c  ongress.  The  details  of 


the  exercise  included  numerous  searches,  engagements,  and  movements. 
The  exercise  started  with  an  assault  on  a  beach  and  terminated  with  the 
withdrawal  of  troops  back  to  the  ships.  The  exercise  was  observed  for 
two  successive  days  (10  hours  each  day).  The  withdrawal  took  place  on 
the  third  day  and  was  not  observed  or  simulated. 


Field  Exercise  Data  Collection 


Seven  observers  were  trained  and  they  collected  the  required 
TWSEAS  performance  information.  Field  exercise  data  were  collected: 
(1)  by  observers  who  traveled  with  mobile  units  and  collected  data  con¬ 
cerning  referee  and  radio  operator  message  processing,  and  (2)  at  the 
control  van  concerning  the  computer  and  controller  message  processing. 


Computer  Runs  Completed 

Input  data,  derived  as  described  in  Siegel,  Leahy,  &  Wolf  (1979), 
were  entered  in  the  required  format  for  simulation  within  the  NETMAN 
model.  One  run  was  organized  to  simulate  TWSEAS  operation  during  the 
first  day  (D-day)  of  the  field  exercise  and  another  run  was  completed  to 
simulate  D-day  plus  one.  Ten  hours  of  operation  were  simulated  each 
day  to  provide  a  total  of  20  hours  of  model  predictions  which  could  be  com 
pared  with  TWSEAS  performance. 

Validation  Analysis 

The  agreement  between  the  model's  predictions  and  the  TWSEAS 
performance  during  the  field  exercise  was  examined  according  to  the 
eight  criterion  variables. 


Thoroughness 


Thoroughness  within  the  model  is  calculated  as  the  number  of  mes 
sages  completely  processed  divided  by  the  number  of  messages  available 
to  be  processed  in  each  hour.  A  tally  was  made  of  the  number  of  times 
that  the  model's  predictions  fell  within  one  standard  deviation  of  the  field 
data  or  within  one  half  standard  deviation  of  the  field  data.  There  were 
20  comparisons  based  on  10  hours  per  day  for  tw  o  days.  Out  of  the  twenty 
comparisons,  nineteen  of  the  models  hourly  predictions  fell  within  .5 
standard  deviation  of  the  field  hourly  data.  According  to  the  sign  test 
(two  tailed),  this  agreement  ratio  (19  out  of  20)  is  statistically  significant 
at  the  .00  1  level.  Since  the  thoroughness  index  is  fundamentally  a  meas¬ 
ure  of  network  throughput,  this  finding  seems  particularly  important. 
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Responsiveness 


Responsiveness,  within  the  model,  is  calculated  as  the  ratio  of 
human  message  processing  time  to  total  message  processing  time.  To¬ 
tal  message  processing  time  includes  human  time  plus  other  time  such 
as  delays  or  waiting  time.  Model  responsiveness  predictions  fell  within 
one  standard  deviation  of  the  actual  TWSEAS  hourly  performance  data  18 
times  out  of  20.  This  agreement  ratio  is  indicated  to  be  statically  sig¬ 
nificant  by  a  two  tailed  sign  test  at  the  .001  level  of  confidence. 


Overall  Effectiveness 

Overall  effectiveness,  within  the  model,  is  calculated  as  a  com¬ 
posite  of  four  individual  effectiveness  components- -thoroughness,  accu¬ 
racy,  completeness,  and  responsiveness. 

The  model's  hourly  predictions  for  overall  effectiveness  were  com¬ 
pared  with  plus  or  minus  one  standard  deviation  of  the  TWSEAS  criterion 
in  a  two  tailed  sign  test.  The  result  was  not  statistically  significant. 
Ratings  of  overall  system  effectiveness  by  actual  controllers  were  sub¬ 
stantially  lower  than  the  model's  prediction.  This  lower  rating  by  the 
controllers  may  have  been  due  to  the  presence  of  equipment  and  radio 
transmission  difficulties,  which  were  bypassed  during  the  exercise  through 
the  use  of  alternative  delivery  methods.  These  difficulties  were  not  sim¬ 
ulated  in  the  model.  Only  messages  which  are  able  to  pass  through  the 
normal  TWSEAS  channel  are  considered  in  the  simulation. 


Total  Message  Processing  Time 

The  TWSEAS  total  message  processing  time  criterion  data  were 
compared  with  the  NETMAN  output  by  means  of  a  matched  pairs  t-test. 
The  resulting^  value  was  .381.  With  an  N  of  20,  this  value  is  not  statis¬ 
tically  significant  (i.  e. ,  the  model  versus  TWSEAS  time  difference  was 
not  significantly  different).  This  finding  supports  a  contention  of  reason¬ 
able  correspondence  between  predicted  and  actual  TWSEAS  total  message 
processing  time.  Since  total  message  processing  time  is  an  especially 
important  aspect  of  the  simulation,  this  finding  reinforces  arguments  fav¬ 
oring  the  utility  of  the  NETMAN  model. 


Operator  Processing  Time 

During  the  field  exercise,  the  time  spent  working  on  each  message 
was  measured  for  each  individual  (of  those  who  were  under  observation). 

1  hose  data  were  averaged  across  operator  type  to  produce  mean  time  per 
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message  per  hour  for  each  operator  type  (referee,  radio  operator,  or 
controller).  The  NETMAN  produces'  parallel  data.  Independent  com¬ 
parisons  were  performed  for  referee  processing  time,  radio  operator 
processing  time,  and  controller  processing  time.  In  each  comparison, 
the  frequency  of  the  NETMAN  prediction  data  falling  within  plus  or  minus 
one  standard  deviation  of  the  mean  of  the  TWSEAS  observational  data  was 
determined.  In  each  case,  20  out  of  20  of  the  model's  predictions  fell 
within  one  standard  deviation  of  the  TWSEAS  criterion.  For  each  posi¬ 
tion,  the  two  tailed  sign  test  was  statistically  significant  at  the  .001  level 
of  confidence.  This  finding  indicates  considerable  agreement  between 
the  NETMAN's  predictions  and  the  TWSEAS  time  criterion  at  all  levels 
of  message  processing.  The  agreement  between  predicted  and  criterion 
time  is  especially  important  since  modeling  manipulations  of  operator 
time  (for  example,  by  varying  operator  proficiency  level  or  through  task 
revision)  can  provide  important  information  to  system  designers  con¬ 
cerning  the  required  number  of  personnel  and  the  required  training  of 
personnel  in  new  systems.  Such  simulations  also  provide  information 
concerning  the  workload  imposed  on  personnel,  and,  therefore,  the 
availability  of  such  personnel  for  additional  responsibilities. 


Transmission  Delay  —  .  _ _ 

Transmission  delay  in  the  TWrSEAS  is  represented  by  the  time  be¬ 
tween  a  first  attempt  to  transmit  a  message  and  the  time  recorded  by  the 
computer  as  the  time  of  message  entry. 

Only  one  overall  transmission  time  output  recording  is  available 
per  simulated  day  from  the  NETMAN  computer  model.  This  one  pre¬ 
diction  time  was  compared  with  the  data  for  each  hour  from  the  TWSEAS 
criterion.  Comparison  of  the  predictions  of  the  NETMAN  model  with  the 
TWSEAS  performance  indicated  that  the  model  data  were  within  .5  stand¬ 
ard  deviation  of  the  criterion  data  in  19  out  of  the  20  comparisons.  This 
agreement  is  statistically  significant  (sign  test)  at  the  .001  level. 


Integration  of  Validation  Data 

Figure  3-1  shows  the  correspondence  between  the  NETMAN  model's 
predictions  and  the  TWSEAS  data  in  terms  of  deviation  from  the  TWSEAS 
data.  The  "x's"  represent  the  mean  model  predictions  across  all  hours, 
while  the  mean  and  standard  deviation  of  the  horizontal  axis  represent  the 
mean  and  standard  deviation  of  all  field  data  points.  Two  of  the  predictions 
agreed  almost  exactly  with  the  respective  criterion--thoroughness  and  to¬ 
tal  processing  time.  Five  additional  predictions--responsiveness,  referee 
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PEST  AVAILABLE  COPY 


Field  Reference 
Number  of  Standard  Deviations 

SD  -3  -2  -1  MEAN  +1  +2 


Thoroughness  [  X  i 

Responsiveness  |  X( 

Effectiveness  !  •  X 

:  ; 

Total  Processing  Time  !  X  i 

Referee  Processing  Time  X  i 

; 

Radio  Operator  Processing  Time  !  X  ■ 

:  : 

Controller  Processing  Time  !  X  i 

Transmission  Delay  !  X  i 


+3 


Figure  3-1.  Model  predictions  versus  field  data. 


processing  time,  radio  operator  processing  time,  controller  processing 
time,  and  transmission  delay,  were  within  one  standard  deviation  of  the 
field  data.  Only  one  of  the  indices  investigated,  overall  effectiveness, 
was  more  than  one  standard  deviation  from  the  mean  of  the  field  data. 

In  this  case,  the  deviation  exceeded  two  standard  deviations.  As  was  in¬ 
dicated,  there  was  a  lack  of  correspondence  between  the  TWSEAS  overall 
effectiveness  as  rated  by  the  controllers  and  overall  effectiveness  as 
measured  by  the  model. 

Table  3-1  summarizes  the  statistics  and  results  of  the  eight  val¬ 
idation  analyses.  The  sign  tests  were  all  structured  to  test  the  degree 
of  agreement  between  model  and  field  data.  Agreement  to  within  one 
standard  deviation  was  found  in  all  cases  except  for  overall  effectiveness. 
A  tptest  was  performed  to  compare  model  and  field  total  processing  time. 
No  statistically  significant  difference  was  found  between  model  predictions 
and  TWSEAS  data  in  this  case. 


Discussion 


The  results  of  the  validation  support  contentions  favoring  the  va¬ 
lidity  of  thcNETMAN  model  relative  to  all  but  one  of  the  diverse  TWSEAS 
criteria  employed.  The  degree  and  magnitude  of  the  diverse  events  in¬ 
volved  in  field  exercises  in  which  a  TWSEAS-like  system  is  involved 
bring  out  clearly  the  need  for  computer  models  in  this  area.  Where 
variable  occurrences  overshadow  design  variables,  the  need  for  im¬ 
provements  is  very  hard  to  evaluate.  Each  field  exercise,  by  its  nature, 
is  expensive  and  provides  only  a  small  sample  of  possible  conditions. 
Within  a  model,  however,  the  effects  of  such  events  can  be  assessed  by 
combining  the  effects  of  many  low  probability  events  through  stochastic 
iteration. 

It  was  reported  that  the  criterion  quality  was  believed  to  be  rea¬ 
sonably  acceptable,  given  the  limited  amount  of  data,  i.  e. ,  the  number  of 
messages  that  the  referees  were  able  to  transmit  from  the  field  over  the 
normal  TWSEAS  system.  (The  small  message  sample  size  represents 
the  primary  limitation  on  the  adequacy  of  the  criterion  data.  ) 

The  measurement  of  the  TWSEAS  performance,  as  performed  by 
the  field  observers,  was  believed  to  bo  highly  reliable.  Although  no 
formal  independent  measures  were  available,  the  use  of  discrete  time 
points  and  message  identifiers  when  coupled  with  the  training  that  was  ad¬ 
ministered,  allow  a  high  degree  of  confidence  in  the  reliability  of  the  data. 
The  exercise  data,  as  collected,  were  highly  objective.  Discrete,  prede¬ 
fined  starting  and  stopping  points  were  identified  for  time  measurements. 
Subjective  evaluations  were  used  only  in  the  case  of  the  controller's  over¬ 
all  effectiveness  evaluation. 
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TWSEAS  performance  data  required  little  or  no  transformation, 
rescaling,  preprocessing,  or  translation  in  order  to  be  used  as  criteria. 

Sensitivity  and  Validation  Summary 

Overall,  the  results  of  the  sensitivity  and  validation  tests  suggest¬ 
ed  substantial  confidence  in  the  NETMAN  simulation  model. 

The  conclusions  reached  by  Siegel,  Leahy,  &  Wolf  (1979)  were: 

1.  The  NETMAN  model  is  adequately  sensitive  over 
a  wide  range  of  parametric  variation. 

2.  The  parametric  variation  introduced  during  the 
sensitivity  tests  produced  results  which  are  rea¬ 
sonable  and  which  possess  proper  directionality. 

3.  The  results  of  the  validity  tests  indicate  that 
NETMAN  can  provide  information  about  exer¬ 
cise  control  system  operation  comparable  to 
that  available  in  a  field  exercise  environment 
such  as  TWSEAS. 

4.  Several  areas  for  model  improvement  are  indi¬ 
cated. 

NETMAN  Enhancements 


Following  the  independent  panel  review,  sensitivity  testing,  and 
validation,  the  combined  experience  on  model  strong  points  and  desired 
enhancements  was  combined.  A  series  of  some  26  potential  enhancements 
was  developed.  In  each  case  a  written  description  of  the  purpose  and 
method  of  the  change  was  prepared.  From  these,  in  turn,  estimates  were 
made  of  the  cost  in  time  units  (scale  1-20)  to  implement  the  change  and  the 
value  (scale  1-10)  of  the  change.  The  selection  of  the  value  rating  was 
determined  on  the  basis  of  estimates  by  the  model's  authors  at  Applied 
Psychological  Services  and  by  a  representative  oT  the  Army  Research 
Institute  who  was  familiar  with  the  model.  Time  estimates  were  pro¬ 
vided  by  the  primary  model  programmer. 

The  2 0  candidates  were  rank  ordered  according  to  the  utility  per 
unit  time  measure.  Those  16  enhancement  candidates  having  the  largest 
measure  were  selected  and  implemented.  These  enhancements  included; 

1.  Reduce  total  message  storage  capabilities 

2.  Provide  positive  execution  statement 

3.  Increase  number  of  task  elements 

4.  lin  reuse  number  of  tasks  allowed 
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5. 

6. 

7. 

8. 
9. 

10. 

11. 

12. 

13. 

14. 

15. 

16. 


Redirect  detailed  output 
Increase  header  clarity 
Develop  user  work  sheets 
Individualize  effectiveness  measures 
Simulate  unexpected  interruptions 
Allow  natural  form  input 
Revise  stress  logic 
Add  error  checks 

Develop  capability  to  bypass  system  levels 
Modify  fatigue  calculation 
Improve  time  synchrony 
Expand  prompting  to  user 


A  revised  user's  Manual  (Leahy,  Siegel,  &  Wolf,  1980)  was  pre¬ 
pared  to  document  model  changes  resulting  from  these  changes. 
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CHAPTER  4 


NETMAN  APPLICATION  TO  EMARS 


During  1979,  Applied  Psychological  Services  developed  a  concep¬ 
tualization  to  two  alternative  military  field  exercise  management  systems. 
The  report  describing  these  system  concepts  (Siegel,  Webb,  Wolf,  & 
Wilson,  1980)  included  summary  concepts,  block  diagrams,  subsystem  de¬ 
scriptions,  personnel  requirements,  manning  requirements,  and  task  re¬ 
quirements  for  each  of  the  two  system  concepts. 

Technological  advances  of  the  last  decade  provided  the  impetus 
for  the  conceptualization  since  these  advances  have  provided  a  capability 
to  revolutionize  completely  the  way  in  which  the  combat  readiness  of 
ground  military  units  may  be  evaluated.  Moreover,  similar  changes  in 
technology  have  changed  ground  combat  per  se  so  as  to  make  new  ap¬ 
proaches  to  combat  effectiveness  and  readiness  testing  imperative.  These 
developments  include  instrumented  training  and  testing  grounds,  use  of 
lasers  for  kill/  hit  assessment,  ruggedized  (portable)  and  miniaturized 
computational  capability,  and  interactive  graphic  displays.  They  offer 
the  potential  for  simulated  battle  under  conditions  of  unprecedented  re¬ 
alism. 


Two  different  levels  of  exercise  control  and  monitoring  systems 
were  defined.  Each  meets  a  different  need.  The  first  concept  and  the 
one  on  which  the  NETMAN  application  described  in  this  chapter  was  bas¬ 
ed  represents  a  modest  exercise  monitoring,  assessment,  and  reporting 
system  (EMARS).  It  does  not  include  exercise  control.  Such  a  system 
could  be  deployed  at  selected  Army  division  level  headquarters  areas  for 
use  in  testing  units  up  to  company  level. 


The  EMARS  Concept 

The  EMARS  (Siegel,  Webb,  Wolf,  dv  Wilson,  1980)  is  conceived  as 
a  computer  assisted  exercise  monitoring  system.  The  EMARS  can  sup¬ 
port  field  exercise  administration  for  exercises  up  to  and  including  the 
company  level.  Its  overall  functions  are  to; 

©  monitor  the  activities  of  units  during  an  exercise 

©  integrate  the  information  required  for  exercise  control 

©  accept  ami  process  field  data  and  information 

®  generate  situational  displays 
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©  assess  performance  and  generate  objective  scores 
for  individual  units  and  combined  military  units  at 
all  tested  levels  by  time  period 

©  suppoi-t  exercises  up  to  four  days  in  duration 

©  provide  multiple  hard  copy  results- -maps,  records, 
scores,  and  referee  comments 

©  be  capable  of  one  day  readiness  between  exercises. 

Application  of  NETMAN  to  EMARS 

The  EMARS  concept  provided  a  "test  bed"  for  evaluating  the  adapt¬ 
ability  or  utility  of  NETMAN  for  evaluating  systems  while  they  are  in  the 
conceptual  stage  of  development.  Quite  obviously,  models  such  as  NET- 
MAN,  are  of  maximum  utility  if  they  can  provide  evaluative  insight  while 
systems  are  in  the  conceptual  stage  of  their  developmental  cycle. 

The  EMARS  accomodates  five  persons:  personnel/ logistics  mon¬ 
itor  (PER/  LOG),  operations  intelligence  monitor  (OPS/1NT),  ^indirect  fire 
and  air  space  manager  (IF/  ASM),  exercise  management  and  information 
controller  (EM1C),  and  exercise  monitoring  officer  charge  (EXMOIC). 
The  first  three  of  these  possess  the  responsibility  for  receiving  and  as¬ 
sembling  information  relative  to  their  respective  interest  areas,  coding 
the  information,  and  entering  the  information  into  the  data  management 
computer  system.  The  final  two  EMARS  personnel,  the  EMIC  and  the 
EXMOIC,  possess  decision  making  responsibility. 

Accordingly, the  PER/ LOG,  OPS/1NT,  and  IF/ ASM  personnel  were 
conceived  as  analogous  to  the  radio  operators  in  the  NETMAN  scheme  and 
the  EMIC  and  the  EXMOIC  were  conceived  as  controllers  in  the  NETMAN 
scheme. 


Referees  are  not  included  in  the  EMARS  concept.  Rather,  infor¬ 
mation  is  assumed  to  arrive  at  the  EMARS  through  a  variety  of  other 
sources.  Accordingly,  the  referee  representation  was  not  considered 
relevant  to  the  present  simulation. 

The  capability  to  bypass  a  system  level  during  message  processing, 
(Hem  13,  page  32)  was  utilized  during  the  EMARS  simulation.  The  system 
level  bypass  feature  was  used  at  the  referee  level  although  message  gen¬ 
eration  was  simulated. 
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Accordingly,  the  EMARS  simulation  involved  three  levels; 


NET  MAN  Model  Level 


RO 


PER/ LOG 
EMARS  OPS/INT 
Position 

IF/  ASM 


Computer  Controller 

Computer  EXMOIC 

EM1C 


This  organization  allowed  full  simulation  of  the  EMARS  including 
receipt  of  field  based  information  by  EMARS'  computer  personnel,  (PER/ - 
LOG,  OPS/INT,  and  IF/ ASM),  entry  of  these  data  by  EMARS'  personnel 
into  a  computer  based  management  system  and  processing  of  these  data  by 
the  computer,  and  action  on  the  basis  of  the  data  by  EMARS'  decision 
making  personnel  (EMIC  and  EXMOIC). 

Stated  alternatively,  the  EMARS'  personnel  described,  their  func¬ 
tions  and  numbers,  were  defined  in  NETMAN  as  two  operating  networks. 

In  each  network  there  were  three  levels;  (1)  PER/ LOG,  OPS/INT,  and 
IF/ASM,  (2)  computer  and  its  systems,  and  (3)  EMIC  and  EXMOIC.  The 
first  network  is  composed  of  the  PER/ LOG,  the  OPS/INT  and  the  EXMOIC. 
The  second  network  is  composed  of  the  IF/ ASM  and  the  EMIC. 

Task  analytic  input  data  for  each  of  the  simulated  EMARS'  positions 
were  prepared  and  seven  NETMAN  parameters  were  selected  for  para¬ 
metric  manipulation- -four  physical  and  three  human  parameters.  These 
parameters,  along  with  their  variation  over  model  runs  in  the  EMARS 
simulation,  are  presented  in  Table  4-1.  The  first  two  physical  factors 
were  concerned  with  the  number  of  messages  generated.  The  number  of 
messages  generated,  of  course,  determines  the  load  on  the  system  and 
the  stress  on  the  simulated  personnel.  The  next  two  physical  factors  were 
concerned  with  the  length  of  the  messages  entering  into  and  exiting  from 
the  system.  The  human  parameters  were  concerned  with  the  proficiency 
of  the  simulated  assigned  personnel  and  their  level  of  aspiration. 

A  total  of  22  operational  c  omputer  runs,  a  baseline  run,  and  a  con¬ 
trol  run  was  completed.  The  operational  runs  involved  a  wide  spread  of 
variations  within  each  parameter.  In  each  run,  only  one  parameter  is 
varied  while  all  others  are  held  at  baseline  value.  The  baseline  value  of 
each  variable  is  indicated  in  a  separate  column  of  Table  4-1.  These  val¬ 
ues  an-  essentially  "average"  input  values  whose  output  may  be;  compared 
with  the  output  produced  by  the  other  parametric  variations  listed. 
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The  control  column  represents  a  previously  completed  simula¬ 
tion  run  for  the  TWSEAS  system  (Siegel,  Leahy,  &  Wolf,  1979).  The 
run  selected  is  the  available  prior  run  which  most  closely  approached 
the  baseline  parameters  of  the  present  simulations. 


Results  and  Implication  of  Application  for  NETMAN 

The  purpose  of  the  application  of  NETMAN  to  EMARS  was  to  al¬ 
low  some  insight  into  the  flexibility,  transportability,  and  generality  of 
the  NETMAN  model.  Application  of  NETMAN  to  EMARS  involved  the  de¬ 
velopment  of  unique  task  analytic  data,  special  input  data,  use  of  the  en¬ 
hanced  NETMAN  model,  and  interpretation  of  the  output  against  the  back¬ 
drop  of  the  EMARS  concept  and  the  TWSEAS  system.  The  preparation  of 
the  required  task  analytic  and  associated  input  data  was  completed  without 
undue  difficulty. 

The  results  favored  contentions  supporting  the  adaptability  of  the 
NETMAN  model.  The  enhanced  NETMAN  model  yielded  a  set  of  data 
which  was  rational  and  which  was  related  to  the  design  characteristics  of 
the  EMARS  system.  The  model  provided  insight  into  system  throughput 
limitations  and  methods  for  correcting  these  limitations.  For  example, 
interpretation  of  the  results  from  the  NETMAN  application  provided  in¬ 
sight  into  the  requirement  for  an  additional  monitor  in  the  EMARS  sys¬ 
tem. 


Comparisons  were  also  possible  between  the  EMARS  system  and 
the  TWSEAS-type  system  by  virtue  of  computer  runs  employing  essen¬ 
tially  parallel  parameters  but  with  different  network  organizations  and 
task  analyses.  The  results  provided  direct  estimates  of  the  extent  of 
superiority  of  EMARS  over  the  TWSEAS-type  system  in  terms  of  such 
systems  effectiveness  indices  as:  overall  effectiveness,  thoroughness, 
responsiveness,  completeness,  and  performance  degradation  as  a  func¬ 
tion  of  message  length  and  frequency.  These  metrics  are  believed  to  pos¬ 
sess  impoilant  implications  for  comparative  evaluation  of  alternate  sys¬ 
tems  and  for  absolute  evaluation  of  an  individual  system. 

Moreover,  in  all  cases,  the  obtained  results  seemed  reasonable 
in  that  they  were  in  a  direction  which  would  be  anticipated  and  the  model 
was  sensitive  to  variation  in  parameters  such  as  message  length,  mes¬ 
sage  frequency,  operator  proficiency,  operator  precision,  and  operator 
level  of  aspiration.  Moreover,  NETMAN 's  sensitivity  to  differences  be¬ 
tween  the  simulated  systems  suggests  that  NETMAN  application  may  rep¬ 
resent  a  reasonable  way  to  test  economically  and  efficiently  the  capabili¬ 
ties  of  suc  h  systems  while  they  are  in  the  conceptual  stage  of  development. 
Mo  mover,  it  seems  that  such  evaluations  may  be  completed  with  some 
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degree  of  precision  and  some  assurance  that  the  results  will  possess 
meaningful  implications  for  equipment  and  personnel  subsystem  design. 

The  results  of  the  earlier  sensitivity  and  validation  tests  (Siegel 
et  al. ,  1979)  of  NETMAN,  along  with  the  current  findings,  combine  to 
suggest  that  NETMAN  is  a  useful  evaluation  tool.  Its  validity  was  pre¬ 
viously  demonstrated  and  contentions  favoring  its  flexibility  and  adapt¬ 
ability  are  supported  by  the  application  to  the  EMARS  concept. 


CHAPTER  5 


SUMMARY,  NETMAN'S  APPLICABILITY,  AND  IMPLICATIONS 


Following  experience  with  MANMOD,  NETMAN  was  developed 
specifically  to  simulate  field  exercise  management  systems,  and  an 
initial  (limited)  sensitivity  test  was  completed.  Then,  the  model  was  the 
subject  of: 

(1)  panel  evaluation 

(2)  sensitivity  testing 

(3)  validation  testing 

(4)  enhancement 

(5)  application  to  the  EMARS  field  exercise 
management  system 


The  initial  test,  as  well  as  each  of  these  five  tasks,  met  with  gen¬ 
eral  success  as  reported  above: 

•  the  initial  sensitivity  tests  indicated  reasonable  output, 
but  identified  some  problem  areas  (see  Chapter  1,  Initial 
NETMAN  Sensitivity  Tests) 

®  an  independent  panel  of  six  experts  raised  22  issues  for 
consideration  which  were  prioritized  (see  Chapter  3,  In¬ 
dependent  Review  Panel)- -and  held  for  later  consideration 

•  a  sensitivity  test  consisting  of  over  60  simulations 
was  conducted.  It  was  concluded  that  the  statistical 
results  were  directionally  logical,  that  the  model 

is  workable,  and  that  the  most  sensitive  (important) 
variables  are  operator  speed,  operator  precision  and 
network  configuration  (sec  Chapter  3,  Sensitivity  Test¬ 
ing  Methods  anti  Results) 

•  model  validation  against  eight  TWSEAS  criteria  show¬ 
ed  (see  Chapter  3,  Integration  of  Validation  Data)  that 
two  of  the  predictors  agreed  almost  exactly  with  the 
criterion  data,  five  additional  predictors  provided  pre¬ 
dictions  which  were  within  one  standard  deviation  of  their 
respective  criterion,  and  one  prediction  was  two  standard 
deviations  from  the  criterion  data 
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•  some  27  candidate  suggestions  for  NETMAN  enhance¬ 
ment  were  summarized  and  prioritized,  17  of  which 
were  implemented  and  documented  (see  Chapter  3) 

•  the  model  was  applied  to  EMAKS  (see  Chapter  4) 
and  it  was  found  that  the  model  is  reasonably  sen¬ 
sitive,  transportable,  and  adaptable. 


However,  there  are  several  aspects  of  the  NETMAN  simulation 
which  a  user  should  understand: 

•  NETMAN  considers  only  those  key  variables 
shown  in  Table  2-2.  As  discussed  in  Chapter  2, 
some  of  these  variables  are  not  easily  modified. 

Although  the  ranges  of  values  are  comfortable  in 
terms  of  systems  simulated  to  date,  they  may 
represent  restrictions  on  the  model's  future  ap¬ 
plicability. 

•  NETMAN  deals  almost  exclusively  with  the  mes¬ 
sage  communications,  and  time  aspects  of  a 
field  exercise  management  system.  For  example, 
environmental  factors  and  equipment  design  are 
not  directly  involved  in  the  simulation. 

•  Use  of  NETMAN  implies  the  use  of  the  built-in  se¬ 
quence  of  message  movement.  Names  of  operators 
and  their  tasks  may  be  varied,  system  levels  may 
be  deleted,  and  the  computer  may  be  exchanged  with 
an  operator  (and  vice  versa).  However,  the  frame¬ 
work  is  otherwise  preconceived. 


Within  these  limitations,  it  has  been  found  that: 

•  parameter  inputs  are  readily  selected  and  varied 

®  the  effort  of  preparation  for  use  of  the  model  and 
the  cost  of  use  is  reasonable 

•  the  printed  outputs  are  readily  understood  and  useful 

•  the  user  documentation  is  comprehensible 

•  the  model  is  adequately  sensitive  over  a  wide  range 
of  parametric  variations 

•  the  test  results  are  reasonable  and  possess  proper 
directionality 
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•  NETMAN  provides  data  on  exercise  control  system 
operation  which,  within  reasonable  error,  predicted 
actual  field  exercise  control  system  performance. 

With  respect  to  the  status  of  NETMAN,  consider  Figure  5-1,  re¬ 
produced  from  Siegel,  Leahy,  &  Wolf  (1979).  Figure  5-1  depicts  the  gen¬ 
eral  sequence  of  steps  in  model  development,  validation,  and  imple¬ 
mentation.  NETMAN  has  progressed  from  the  earliest  stages  of  concep¬ 
tual  design  as  indicated  in  Figure  5-1,  through  the  various  stages  to  the 
point  where  (subject  to  its  limitations)  it  is  now  ready  to  enter  the  status 
of  "Decision  Aiding.  " 

As  such  it  is  available  for  simulations  of  field  exercise  manage¬ 
ment  systems  in  which: 

1.  predictions  of  the  effect  of  operator  speed  and 
precision  on  system  performance  are  of  concern 

2.  message  length,  message  frequency,  and  queue 
length  plan  an  important  role 

3.  several  levels  are  involved  and  each  involves  a 
set  of  message  processing  tasks 

4.  the  parameters  of  system  size  are  within  the 
model  limits  given  in  Table  2-2. 

Recommendations 

Given  the  limitations  and  restrictions  cited  above,  the  NETMAN 
model  is  now  believed  to  be  adequately  tested,  evaluated,  and  documented. 
The  model  is  believed  to  possess  utility  for  simulating  field  exercise  man¬ 
agement  systems.  The  questions  of  "why,  "  "how,  "  and  "when"  to  use 
NETMAN,  as  summarized  in  Appendix  A,  are  applicable  and  pertinent. 

®  NETMAN  can  now  be  used  as  an  evaluative  tool, 
particularly  in  the  comparative  evaluation  of  sys¬ 
tem  alternatives  which  emphasize  the  message 
oriented  portions  of  field  exercise  systems.  Feed¬ 
back  from  its  use  should  be  monitored  in  the  im¬ 
mediate  future  by  model  development  as  well  as 
user  personnel  to  ensure  proper  use  as  well  as  to 
evaluate  potential  improvements  in  the  model  as 
indicated  by  operational  use. 
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I’if>ui  e  5-1.  Sequence  in  model  development. 
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•  The  model,  program,  and  user  documentation 
are  now  available  to  appropriate  user  agencies.  A 
training  course  for  NETMAN  users  should  be  con¬ 
sidered  as  a  technique  for  application  support. 

•  NETMAN  constitutes  a  design  decision  aid  which  is 
particularly  useful  to  system  designers  in  gaining 
insight  into  the  answers  to  "what  if"  questions  rel¬ 
ative  to  system  structure  and  organization. 

Due  to  the  model  response  to  the  parametric  variations  of  the 
sensitivity  tests  and  the  agreement  between  the  model  and  the  TWSEAS 
criterion  data,  a  substantial  degree  of  confidence  may  be  placed  in  in¬ 
dications  derived  from  the  NETMAN  computer  model.  Moreover,  the 
model  was  implemented  at  a  relatively  low  cost.  Future  exercise  con¬ 
trol  system  design  would  benefit  from  early  test  through  the  use  of  NET 
MAN.  Moreover,  the  NETMAN  model  may  be  used  to  determine  or  con 
firm  personnel  allocation,  effects  of  personnel  proficiency,  effects  of 
various  operator  characteristics,  network  configuration,  and  the  like  in 
present  network  oriented  exercise  control  systems. 
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Inherent  in  any  computer  simulation  concept  is  the  understanding 
that  considerable  savings  can  be  achieved  by  substituting  model  exercises 
for  actual  system  test.  Such  savings,  of  course,  are  predicated  on  a  de¬ 
monstration  that  adequate  agreement  can  be  achieved  between  a  model, 
such  as  NETMAN,  and  actual  system  operation.  Also  to  the  computer 
model's  credit  is  the  aspect  of  relative  time  required  to  yield  viable  re¬ 
sults  when  compared  with  actual  system  exercise.  The  initial  data  col¬ 
lection  and  preparation  for  a  model  simulation  may  be  extensive  but,  once 
completed,  the  results  of  parametric  variations  may  be  obtained  in  very 
short  times.  It  should  be  possibLe  for  a  model  to  yield  results  in  a  few 
hours  which  would  require  weeks  of  time  and  teams  of  personnel  in  the 
actual  exercise  situation.  Moreover,  due  to  the  interactive  (remote  ter¬ 
minal)  capability,  the  completion  of  such  stochastic  simulations  is  highly 
convenient.  Other  advantages  of  models  in  general  over  actual  system 
exercises  are; 

®  exercising  a  model  is  less  costly 

®  fewer  personnel  are  involved 

•  models  are  independent  of  uncontrollable 
conditions 

®  models  do  not  expose  personnel  to  danger 
or  accidents 

®  models  do  not  expose  equipment  to  damage 
®  convenience 

Note  also  that  the  more  complex,  costly  or  large  scale  the  operational 
system,  the  more  dominant  these  relative  advantages  of  simulation  mod¬ 
eling  become. 

In  addition,  the  simulation  alternative  offers  the  capability  to  con¬ 
sider  and  evaluate  tire  impact  of  new  anticipated  equipment,  different 
speeds  and  numbers  of  communication  lines,  as  yet  unauthorized  operator 
sequences/ procedures,  and  system  loads. 

Proper  use  of  NETMAN  will  allow  the  identification  of  system  mod¬ 
ifications  which  afford  improvment  to  the  system.  By  testing  the  effects  of 
many  types  of  possible  system  modification,  true  optimization  of  the  sys¬ 
tem  can  be;  approached. 
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How 


The  model  can  be  of  use  in  the  area  of  personnel  planning  in  a 
number  of  ways.  Primarily  the  areas  of  personnel  requirements,  train¬ 
ing,  and  workload  can  be  assessed.  Personnel  characteristics  such  as 
speed,  precision,  le^el  of  aspiration,  and  stress  threshold  are  a  part  of 
the  model's  input  data.  By  manipulating  the  personnel  characteristics 
and  examining  the  resulting  system  performance,  the  effects  of  varia¬ 
tions  in  personnel  parameters  can  be  assessed. 

The  model's  output  includes  such  parameters  as  percentage  of 
time  worked  and  mean  stress  level  during  the  hour.  By  evaluating  these 
values  the  design  of  work  schedules  which  produce  reasonable  workloads 
and  acceptable  stress  levels  can  be  implemented. 

NETMAN  can  be  used  to  investigate  the  effects  of  equipment  modi¬ 
fications  or  substitutions  on  system  performance.  The  effects  of  equip¬ 
ment  are  included  in  the  model  through  the  task  analysis  for  each  operator. 
The  task  analysis  includes,  among  other  things,  the  time  required  for  op¬ 
erating  each  equipment. 

In  the  evaluation  of  equipment,  however,  we  note  that  NETMAN 
does  not  actually  test  equipments--only  the  equipments'  capacity  to  sup¬ 
port  the  performance  requirements  is  considered  by  the  simulation.  The 
reliability  or  accuracy  of  any  particular  type  of  equipment  must  be  tested 
separately  and  its  performance  characteristics  are  incorporated  into  the 
task  analytic  input  data.  The  effect  of  the  equipment  on  system  perform¬ 
ance  can  then  be  evaluated. 

When 


There  are  three  major  periods  in  system  development  in  which 
models  such  as  MANMOD  and  NETMAN  can  be  used.  They  can  be  used  as 
an  aid  in  the  initial  design  of  a  system,  as  a  testing  device  in  the  appraisal 
of  system  quality,  and  as  an  evaluation  tool  for  system  redesign  or  retrofit. 
The  use  of  the  model  in  all  three  cases  would  be  basically  the  same;  how¬ 
ever,  the  degree  of  accuracy  of  the  input  data  will  increase  as  more  pre¬ 
cise  use  data  are  available.  As  the  accuracy  of  the  model  input  data  in¬ 
creases,  the  accuracy  of  the  predictions  of  the  model  can  be  expected  to 
increase  proportionally. 
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appendix  b 


MANMOD  Model  Chronology 

An  initial  report  was  published  (Siegel,  Wolf,  &  Leahy,  1973) 
which  described  the  MANMOD  digital  computer  model  for  simulation  of 
message  processing  tasks  performed  by  field  army  personnel  during  a 
Tactical  Operations  System  (TOS)  mission.  That  report  defined  vari¬ 
ables,  described  a  computational  logic  flow  which  integrated  the  vari¬ 
ables  into  a  coherent  digital  simulation  model,  and  presented  the  results 
of  initial  tesis  of  model  sensitivity. 

As  a  result  of  model  utilization  and  testing,  a  number  of  avenues 
for  model  improvement  became  evident: 

•  extension  to  allow  an  experimenter,  seated  at  a  CRT 
display  terminal,  to  initiate,  control,  and  monitor 
results  from  simulation  runs.  The  MANMOD  model 
was  extended  to  allow  the  experimenter  to  interact 
with  the  program  and  make  online  changes  such  as: 
parameter  modification,  assignments  of  personnel 
and  the  tasks  they  perform,  and  characteristics  of 
messages  and  message  frequency. 

•  extension  to  allow  one  or  more  subjects,  seated  at 
CRT  display  terminals,  to  participate  in  a  simulated 
TOS  exercise,  by  performing  specific  TOS  tasks,  se¬ 
lected  as  not  best  allocable  to  digital  simulation,  while 
the  model  simulates  the  balance  of  the  tasks. 

The  MANMOD  was  extended  so  as  to  allow  "hybrid"  simulation  in 
a  man-computer  interactive  manner.  In  this  "hybrid"  mode  of  interactive 
simulation,  a  man  sitting  at  a  cathode  ray  tube  terminal  was  assigned  the 
performance  of  those  tasks  which  are  best  not  treated  through  digital  sim¬ 
ulation  while  the  computer  simulates  the  performance  of  tasks  which  dig¬ 
ital  simulation  treats  well.  The  simulation  is  termed  "hybrid"  since  the 
human  can  be  considered  to  be  an  analog  device  which  performs  in  inter¬ 
action  with  the  digital  logic  of  a  high  speed  computer.  The  end  result 
was  an  interactive  simulation  which  takes  advantage  of  the  time  compres¬ 
sion  of  the  digital  computer  and  allows  inclusion  of  the  flexibility  of  the 
human  components  in  a  man/  machine  system.  The  tasks  to  be  performed 
by  the  on-line  subject  are  prespecified  by  the  person  charged  with  sim¬ 
ulation  c  onduct  and  can  include  such  tasks  as  cathode  ray  tube  monitoring, 
date  entry,  data  comparison,  counting,  verifying,  and  decision  making. 
Figure  B-l  presents  the  data  flow  in  this  computer,  experimenter,  sub¬ 
ject  interactive  mode. 
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Figure  B-1.  System  flow  for  the  computer  experimenter, 
subject  interactive  mode- 
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•  Strengthening  of  the  MANMOD  model  by  incorpo¬ 
rating  a  number  of  model  elaborations  including: 

(1)  extension  of  the  number  of  shifts  which  can  be 
simulated,  (2)  revision  of  the  calculation  of  per¬ 
formance  effectiveness,  (3)  extension  to  allow  the 
generation  of  more  than  one  TOS  message  from  a 
single  system  input  message,  (4)  correction  in  the 
calculation  of  the  number  of  undetected  errors,  (5) 
modification  of  the  calculation  of  operator  stress 
to  allow  consideration  of  the  effects  of  message 
priority  on  stress,  and  (6)  improvement  in  the  con¬ 
tent  and  format  of  the  printed  output. 

In  December  1973,  a  report  (Siegel,  Wolf,  Leahy,  Bearde.  & 
Baker,  1973)  described  the  implementation  of  the  features  into  the  pre¬ 
viously  developed  model. 

The  effects  of  the  modifications  on  the  TOS  simulation  model's 
output  were  tested  through  a  set  of  sensitivity  tests.  The  results  (also 
reported  in  Siegel,  Wolf,  Leahy,  Bearde,  &  Baker,  1973  suggested  that 
the  modifications  produce  reasonable  output  (i.  e. ,  in  conformity  with 
logical  expectation). 

To  achieve  additional  utility,  the  computer  program  representing 
the  original  model  was  modified  and,  as  a  result  two  new  computer  pro¬ 
grams  were  developed.  The  first  program  was  a  revision  of  the  basic 
model  containing  improvements  only  and  did  not  contain  interactive  fea¬ 
tures.  This  was  termed  the  "Strengthened  Model,  "  which  continued  to 
be  called  MANMOD.  The  second  was  an  extended  model  which  incorpo¬ 
rated  the  changes  required  for  the  interactive  features  including  both  the 
experimenter  and  subject  interaction,  as  well  as  the  model  improvements 
included  in  the  "strengthened  model"  (with  the  exception  of  shift  extensions, 
which  are  impractical  for  interactive  runs).  This  was  called  the  "inter¬ 
active  Model.  " 

A  second  part  of  the  subject  interactive  mode  allowed  the  comput¬ 
er  to  call  the  subject  interactive  data  from  disk,  average  it  to  provide 
new  human  performance  data,  and  then  use  these  data  in  a  new  simula¬ 
tion  which  may  be  stochastically  repeated  across  many  varied  iterations. 
The  results  from  such  a  stochastic  combination  of  events  across  many 
iterations  have  been  found  to  provide  a  more  consistent,  representative, 
and  replicable  picture  of  simulated  events. 

During  1975,  a  more  useful  and  more  powerful  version  of  the  MAN- 
MOD  was  developed  for  simulating  the  TOS  network.  This  work  included; 


55 


•  allowing  for  automatic  transfer  of  data  between 
MANMOD  and  complementary  simulation  models 
called  CASE  and  SAMTOS 

•  modifying  the  MANMOD  so  that  it  can  be  run  on 
the  U  1108  computer  system 

•  increasing  the  verisimilitude  of  the  model  through 
incorporation  of  error  message  evaluation  and  in¬ 
terruption  features. 


In  this  program  the  MANMOD  was  coupled  with  the  CASE  and 
SAMTOS  models  to  allow  the  strongest  features  of  each  to  be  exercised 
in  an  integrated  TOS  simulation  and  to  strengthen  the  MANMOD  so  as  to 
allow  greater  simulation  fidelity.  The  coupled  CASE  and  MANMOD  com¬ 
puter  simulation  may  be  performed  on  the  U  1108  system  and  the  results 
entered  directly  to  SAMTOS.  This  approach  achieved  the  desired  cou¬ 
pling  without  modifying  SAMTOS  in  any  way.  Modification  of  the  SAMTOS 
program  was  considered  to  be  undesirable  because  SAMTOS  is  written  in 
a  special  language  and  because  certain  subroutines  in  the  model  are  not 
the  sole  property  of  the  government.  Accordingly,  the  cost  of  such  a 
modification  would  exceed  the  benefit  to  be  obtained  from  direct  integra¬ 
tion. 


In  the  coupled  form,  the  message  delay  data  generated  by  a  re¬ 
vision  of  CASE  are  employed  as  input  to  the  MANMOD.  The  MANMOD 
then  employs  these  data  within  its  simulation  of  the  message  processing 
by  the  system  operators.  The  output  of  the  combined  simulation  is  in  a 
form  that  it  can  be  entered  directly  into  SAMTOS,  which  simulates  fur¬ 
ther  equipment  delay  aspects. 

The  results  of  the  sensitivity  tests  indicated  that  the  message 
processing  time  output  is  sensitive  to  variation  of  the  input  parameters 
which  directly  affect  this  logic.  All  tests  completed  supported  the  con- 
tentionthat  these  modifications  yield  an  output  in  the  anticipated  direction. 

When  the  MANMOD  itself  and  the  coupled  models  are  considered, 
there  seems  to  be  available  the  ability  to  simulate  and  investigate  the  ef¬ 
fects  of  varying  items  such  as  qualitative  and  quantitative  aspects  of  the 
manning,  message  load,  equipment  characteristics  and  delaytimes,  mes¬ 
sage  type,  and  watch  length  on  system  effectiveness.  Moreover,  the  out¬ 
put  will  state  not  only  that  a  given  effectiveness  level  was  or  was  not 
achieved  but  also  what  contributed  to  any  failures  noted.  The  output  de¬ 
tail  can  provide  insights  relative  to  equipment  design  modification,  train¬ 
ing  requirements  and  objectives,  personnel  requirements,  and  tradeoffs. 
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APPENDIX  C 


MANMOD  AND  NETMAN  REPORTS 
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